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implementation— and the escorted trial technique used for evaluation 
during the design phases. An example of the evaluation procedure is 
provided, noting the particular emphasis on four criteria: appeal, 
comprehensibility, accessibility, and responsiveness. Finally, the 
implementation formative evaluation process — which utilized 
videotapes of users, touchscreen records from the computer, and 
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Introduction 

This paper describes the role of formative evaluation in the development of a 
museum videodisc project entitled "Earth Over Time." The term, formative 
evaluation," refers to trying out exhibit materials during their formation in 
order to maximize the appeal and effectiveness of the final exhibit. 

"Earth Over Time" is an earth sciences videodisc sponsored by 15 museums of 
the Interactive Video Science Consortium and produced by Digital Techniques, 
Inc. In general, the activities and content of the disc encourage acquisition 
of the concept that the earth is constantly moving and changing and is not 
simply a solid ball. The videodisc covers plate tectonics, volcanoes, 
earthquakes, sea floor exploration, and shoreline erosion. 



Exhibit Design Problems 

Museums are particularly difficult environments for which to develop effective 
informal learning materials. Museum audiences range widely in age, bring a 
variety of interests and levels of background knowledge, utilize different 
learning styles, and approach computer-based exhibits with unequal experience 
in technology. Exhibits are often expected to maintain the interest of repeat 
visitors and to hold the interest of a group audience, either in a family or a 
class. Moreover, museums present stimulating exhibits that compete 
simultaneously for the visitor's attention, and in some cases, the 
environment can be noisy and distracting. Thus, the informal environment of 
the museum is a challenge for designers. 

To meet some of these environmental demands, the videodisc designers and 
consortium advisors of "Earth Over Time" made a series of design decisions. 
The audience was targeted as 10 to 12 year old children who have come to the 
museum with other children and their parents or teachers. To interest a range 
of visitors, scripts used narrators varying in age and gender and presented a 
human interest viewpoint as well as a direct scientific perspective. 
Activities and information segments were short and varied. To accommodate 
different learning styles, a variety of activity formats were utilized: game, 
simulation, self-paced discovery learning, and receptive expository learning. 

However, the remaining design problems did not have easy solutions. There was 
no basic research that indicated what content and design features would appeal 
most to our audience. And although a touch screen system is physically easy 
to use, there was very little research that would guide screen design to 
assure that users could readily access information, finally, it was not 
clearly apparent how to meet the expectations and needs of the target audience 
nor how to script so that they could understand the scientific information. 
Thi.s, testing the disc with the target audience was critically important to 
designing an effective exhibit. 
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The Continental Puzzle: Pangea (One Earth) . 

Scientists believe that all the continents were once connected,' over 200 
million years ago. 

See if you can connect them now. You can move a continent by 
touching it and sliding it across the screen. 
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For all the activities, we looked at four issues in the data, although only 
the "Continental Puzzle" is discussed as an example below: 

Appeal . Were the activities fun and interesting? The "Continental Puzzle" 
received a very positive response, as users huddled around the pieces and got 
involved in discussions as to where the pieces should move. 

Comprehensibility . Did users understand the information available? Did the 
activity arouse inquiry and curiosity? The puzzle activity elicited a great 
deal of content discussion, but users wanted a hint or clue, because their 
ability to produce a close to correct answer was limited to logic or previous 
knowledge. 

Accessibility . Did users know what to do? Did they know what happened if 
they touched the screen "buttons"? Did they know how to get to another place? 
Everyone understood the concept of moving the pieces around until they fit, 
but no one noticed the "Compare with scientists' view" button, because their 
arms and hands covered it up at the bottom of the screen. 

Responsivenesr . Are the activities what users expected? What kind of 
information and feedback do they want or need? What do users want to be able 
to do? Respondents expected the continents to fit together exactly as puzzle 
pieces do. In terms of feedback, they wanted to be able to move the pieces 
while making a comparison with the correct answer or hint, and they wanted the 
hint to be short. 

Based on this feedback, the "Continental Puzzle" activity was redesigned, as 
were the other activities in the high level design. In the escorted trial 
technique for the detailed design, the researcher again walked parents and 
children through the paper design pages, acting as if she were the computer 
and videodisc. Paper screens and short motion video clips were presented 
while providing oral narration of what visitors would hear. 

To continue our example of the "Continental Puzzle" activity, the detailed 
design is presented on the next page. Again paper continents were provided to 
simulate what would happen on the touch screen. In this version, Africa 
remained immobile, and all continent pieces were to approach Africa. Touching 
the "Get a Hint" button revealed a black shadow of Pangea underneath the 
continents ~ respondents could continue to move their pieces during the five 
seconds that the hint was available. Touching "Take a Peek" showed a small 
completed puzzle of Pangea in the upper righthand corner. "Start Over" 
repositioned the continents. The researcher gave feedback for positioning of 
the cut-out paper continents by saying "beep" when the pieces were correct and 
remaining silent for incorrect positioning. 

The same issues of appeal, comprehensibility, accessibility, and 
responsiveness were considered in the analysis for the detailed design. 
The puzzle appeal was rated higher for the revised version than for the first 
version; both parents and students liked the challenge and fun of putting the 
puzzle together. A large percentage of participants gained a simple 
definitional understanding of Pangea from doing this activity. Most 
respondents did not use the clue of "peek," and many suggested that the clues 
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The Continental Puzzle: Pangea (One Earth) 

Scientists believe that all the continents were once connected 
over 200 million years ago. See if you can connect them now' 
Move a continent by touching it and sliding it across the screen 




gave too much information for too long. The audio "beep" was understood by 
the majority of respondents, although some thought that it meant that time was 
up or that their move was incorrect. The audio feedback was ambiguous. 

The puzzle activity was revised, guided by the evaluation findings. In our 
example of the "Continental Puzzle/ 1 "Get a Hint" was retained whereas "Take a 
Peek" was dropped (see screen on the next page). The feedback mechanism was 
changed so that when a continent was moved into correct position next to 
immobile Africa, its color changed to match Africa and it remained fixed. 
When all the continents were connected properly, the audio would congratulate 
the user. The other "Earth Over Time" activities were ai3o revised and the 
videodisc was pressed and programmed. 

Thus, the escorted triai technique with paper designs gathered cost-effective 
and timely feedback from the target audience. Changes could be made easily 
without repressing discs or reprogramming software. The formative evaluation 
was an inexpensive way of increasing the potential that the exhibit would be 
user-friendly, appealing, and effective. However, even though paper screens 
can mimic the interactivity of the product, they dc not allow testing of the 
actual touch interface. This has to be done with the hardware itself when it 
is implemented in the museum environment. 



Implementation Formative Evaluation 

The data gathered in the museums for "Earth Over Time" included videotapes of 
users, touchscreen records from the computer, and interviews of a sample of 
users. These data are in the process of being analyzed now and will answer 
questions about who used the disc, how it was used, what did they like and 
learn, and did they have any problems. Fine-tuning of the software program 
will occur in response to the museum observations. 

Conclusion 

Formative evaluation allows the target audience to play the role of designer, 
thereby increasing the likelihood that museum visitors will find the product 
appealing, educational, and easy to use. For videodisc projects, the escorted 
trial technique makes it feasible to gather useful and valid diagnostic data 
from target audiences using early paper designs, before major expenditures are 
made. 

In this particular project, the museum audience, through tha evaluation 
process, played two other roles. One was as a mediating voice in the 
decision-making of the consortium. The consortium involved 15 museums, each 
of which had its own agenda in terms of content and design* A number of 
decisions were made and deadlocks resolved by considering the audience's 
viewpoint as represented in the evaluations. 

A final role of the audience was helping to define the allocation of 
resources. With a tight budget, cutbacks were designed to retain what 
the audience found appealing and eliminate the confusing and unappealing 
segments. 
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The Continental Puzzle: Pangea 

Scientists believe that all land was once connected, over 200 million years ago. 
See if you can connect the pieces now. Move a land mass by touching it and 
sliding it across the screen. 
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